Systems, methods, and media for detecting content change in a streaming image system

ABSTRACT

Systems, methods and media for transitioning detecting content change in a streaming image system are disclosed. One embodiment provides a method for detecting a content change between image frames in a streaming image system. Embodiments may include selecting a change detection algorithm from a plurality of change detection algorithms and comparing a first image frame of an image stream with a second image frame of the image stream using the selected change detection algorithm. Embodiments may also include, in the event of detecting a content change between the first image frame and the second image frame, generating an indication of the content change. Further embodiments may include selecting the change detection algorithm based on a user&#39;s selection or system capabilities. Other further embodiments may include transmitting the indication of the content change.

FIELD OF INVENTION

The present invention is in the field of streaming image systems. Moreparticularly, the present invention relates to systems, methods andmedia for detecting content change in a streaming image system.

BACKGROUND

The ability to stream images or other content from a server to multipleclients is a quickly-growing need. Multi-media applications that utilizestreaming images continue to increase in popularity and include videogames, navigation software, streaming movies or video, and the like.These applications, however, often are network-resource intensive andresult in bandwidth bottlenecks and network slowdowns when contentproviders use them to distribute content, particularly to large numbersof users. As the popularity of streaming image applications continues toincrease, the network performance problems associated with them will beexacerbated.

To reduce the impact of streaming image content on a network, contentproviders often compress their images before transmission. The clientsystem must then decompress the image upon receipt before displaying theimage to a user. Depending on the level of compression, network trafficcan be significantly decreased by utilizing compression. One compressionscheme for video images is motion-JPEG which extends the JointPhotographic Experts Group (JPEG) digital image compression standard tovideos by encrypting each frame in the JPEG format. The JPEG groupcreated the ISO/IEC International Standard 10918-1 ITU-T RecommendationT-81 (hereinafter ‘JPEG’) to create a decoding/encoding standard. JPEGand Motion-JPEG are lossy compression standards and thus information islost during the compression process. Motion-JPEG provides good per-framecompression levels but some of its compression steps, such as Huffmancoding, are not always necessary and can slow performance.

The Moving Pictures Experts Group (MPEG) created another family ofcompression standards that include MPEG-1, MPEG-2, and MPEG-4 (ISO/IECInternational Standards 11172, 13818, and 14496, respectively). The MPEGworking group designed the MPEG standards to work for multi-mediastreaming and utilize block-based motion compensated prediction (MCP) toassist in compression. For many applications, MPEG improves upon theperformance of motion-JPEG. For interactive streaming imageapplications, however, MPEG is not optimal. MPEG requires a server togenerate multi-frame movies to achieve good compression levels, makingit less useful for interactive applications that have frame-by-frameinteractions. Instead, MPEG is designed and optimized for streamingpredictable content, such as movies or other videos, to client devices.

Interactive streaming image systems provide significant challenges tocontent providers desiring to distribute content from a server tomultiple clients. Interactive streaming image systems typically receiveuser input for each frame so that each image frame is customized basedon the latest user information. A map-based application, for example,might provide an image frame based on user position and heading so thatthe application could create an image showing the user what they wouldsee at that position and heading. In another example, an applicationthat displays a virtual view of what is behind an automobile may baseits image on the current position, direction, and speed of theautomobile. Because each frame must be recalculated based on newinformation, MPEG does not provide an efficient method as it does notachieve its best compression rates when working with single frames.Similarly, motion-JPEG does not provide any advantage when used withinteractive streaming image systems as it applies a compression methodthat may be too resource-intensive for each image frame. There is,therefore, a need for an effective mechanism for managing a streamingimage system, particularly when the streaming image system isinteractive.

SUMMARY

The problems identified above are in large part addressed by systems,methods and media for detecting content change in a streaming imagesystem. One embodiment provides a method for detecting a content changebetween image frames in a streaming image system. Embodiments mayinclude selecting a change detection algorithm from a plurality ofchange detection algorithms and comparing a first image frame of animage stream with a second image frame of the image stream using theselected change detection algorithm. Embodiments may also include, inthe event of detecting a content change between the first image frameand the second image frame, generating an indication of the contentchange. Further embodiments may include selecting the change detectionalgorithm based on a user's selection or system capabilities.

Another embodiment provides a machine-accessible medium containinginstructions effective, when executing in a data processing system, tocause the system to perform a series of operations for transitioningcompression levels in a streaming image system. The series of operationsgenerally includes detecting a content change between image frames in astreaming image system. Embodiments may include a series of operationsfor selecting a change detection algorithm from a plurality of changedetection algorithms and comparing a first image frame of an imagestream with a second image frame of the image stream using the selectedchange detection algorithm. Embodiments may also include a series ofoperations for, in the event of detecting a content change between thefirst image frame and the second image frame, generating an indicationof the content change. Further embodiments may include a series ofoperations for selecting the change detection algorithm based on auser's selection or system capabilities.

One embodiment provides a content change detector system. The system maygenerally include a component interface module for receiving an imageframe to be compared and a plurality of change detection modules eachhaving an associated change detection algorithm, where the changedetection modules are adapted to compare the received image frame with aprevious image frame using its associated change detection algorithm.The system may also generally include an algorithm selector incommunication with the component interface module to select one of theplurality of change detection algorithms for comparing the receivedimage frame with the previous image frame.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects and advantages of the invention will become apparent uponreading the following detailed description and upon reference to theaccompanying drawings in which, like references may indicate similarelements:

FIG. 1 depicts an environment for a system for streaming images from aserver to a plurality of clients and detecting content changes in theimage stream according to one embodiment;

FIG. 2 depicts a content change detector of the interactive imagestreaming system of FIG. 1 according to one embodiment;

FIG. 3 depicts an example of a flow chart for generating, encoding, andtransmitting a new image frame according to one embodiment;

FIG. 4 depicts an example of a flow chart for receiving user input andreceiving and displaying an image frame to a user according to oneembodiment; and

FIG. 5 depicts an example of a flow chart for selecting a changedetection algorithm and comparing the content of successive image framesaccording to one embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

The following is a detailed description of example embodiments of theinvention depicted in the accompanying drawings. The example embodimentsare in such detail as to clearly communicate the invention. However, theamount of detail offered is not intended to limit the anticipatedvariations of embodiments; but, on the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the present invention as defined by the appendedclaims. The detailed descriptions below are designed to make suchembodiments obvious to a person of ordinary skill in the art.

Systems, methods and media for detecting content change in a streamingimage system are disclosed. One embodiment provides a method fordetecting a content change between image frames in a streaming imagesystem. Embodiments may include selecting a change detection algorithmfrom a plurality of change detection algorithms and comparing a firstimage frame of an image stream with a second image frame of the imagestream using the selected change detection algorithm. Embodiments mayalso include, in the event of detecting a content change between thefirst image frame and the second image frame, generating an indicationof the content change. Further embodiments may include selecting thechange detection algorithm based on a user's selection or systemcapabilities. Other further embodiments may include transmitting theindication of the content change.

The disclosed system may advantageously choose between multiple changedetection algorithms based on user input, system capabilities, or otherfactors. The disclosed system may then apply the selected changedetection algorithm to successive image frames to determine if there isa difference in content between the two image frames. This may allow fora pluggable architecture that accepts any type of change detectionalgorithm and allows the user (or the system) to select the algorithmthat best suits the situation. As new techniques are developed they maybe added to the available change detection algorithms, which may providefor a flexible and adapted system.

The disclosed content change detection system may advantageously be usedwith a compression smoothing system. In one embodiment, upon receivingan indication from the change detection system that a change in contentbetween successive image frames has occurred, the compression smoothingsystem may transition the compression level of image frames from onelevel to another and use a smoothing heuristic to make such changedcompression levels acceptable to a user. This may allow a server toconserve bandwidth by smoothly increasing the compression level whenonly a lower quality image is needed and likewise smoothly decreasingthe compression level when a higher quality image is needed. For aninteractive image streaming system, the server may smoothly providevarying levels of compression depending on the user's activity andinteraction with their environment. The disclosed content changedetection system may also be useful for streaming image systems that arenot interactive or any other image systems where detecting a change incontent between frames is useful.

While specific embodiments will be described below with reference toparticular configurations of hardware and/or software, those of skill inthe art will realize that embodiments of the present invention mayadvantageously be implemented with other substantially equivalenthardware and/or software systems.

Turning now to the drawings, FIG. 1 depicts an environment for a systemfor streaming images from a server to a client and detecting contentchanges in the image stream according to one embodiment. In the depictedembodiment, the interactive image streaming system 100 includes a server102 and a client 106 connected via a network 104. In an interactiveimage steaming system, the client 106 may transmit client information tothe server 102 via network 104 and the server 102 may then generate anew image frame based on the client information. After the server 102generates the new image frame, it may then encode the image frame at adefined compression level and transmit the encoded image frame and anindication of the compression level used to encode the image frame tothe client 106. The client 106 may then decode the image frame based onthe compression level and display the image to a user. Using theinteractive image streaming system 100, the client 106 and server 102may thus work together to generate each image frame of a streaming imagestream based on client information from each cycle. In response todetecting changes in content in the stream, the interactive imagestreaming system 100 may advantageously choose an appropriate level ofcompression (as will be described in more detail subsequently) based onthe client information, the differences in content between successiveimages, and other information, which may result in more efficient use ofbandwidth and processing. A user (or system 100) may advantageouslychoose a change detection algorithm to determine changed content betweensuccessive images from a list of change detection algorithms based onuser preferences, system performance information, etc.

The server 102, the one or more clients 106, and any or all of theircomponents may execute on a general or special purpose computer systemsuch as one or more of a personal computer, workstation, server,mainframe computer, notebook or laptop computer, tablet PC, desktopcomputer, portable computer system, PDA, set-top box, mobile phone,wireless device, or the like. The computer system may, in oneembodiment, include a central processing unit (CPU) and associatedvolatile and non-volatile memory, including random access memory (RAM)and basic input/output system read only memory (BIOS ROM), a systemmonitor, a keyboard, one or more flexible diskette drives, a CD-ROMdrive, a fixed disk storage drive (also known as a “hard drive” or “harddisk drive”), a pointing device such as a mouse, and an optional networkinterface adapter, all electrically connected using a motherboard orsystem planar to electrically connect these components together. Theserver 102 in one embodiment may be an International Business MachineCorporation (IBM®) eServer or similar server having one or moreprocessors, or threads of processors, executing software and/or one ormore state machines coupled with data storage devices such as RAM, readonly memory (ROM), flash memory, compact disk drives, hard drives, andthe like. The client 106 in one embodiment may be a wireless devicehaving a processor coupled with data storage devices and user input andoutput devices, such as a keypad and display screen. Alternatively, awireless handheld device may act as a server 102 to stream images to aclient 106 computer system via network 104.

Network 104 may be any type or combination of data communicationschannels, such as the Internet, an intranet, a LAN, a WAN, an Ethernetnetwork, wireless network, etc. The Internet or other public network maybe particularly useful as network 104 when a centrally located server102 is transmitting streaming image content to a plurality ofdistributed clients 106 as communications between these will befacilitated. Similarly, the Internet or other public network may alsosuited as network 104 when a plurality of distributed servers 102 streamimage content to a central client 106. Some or all of the servers 102 orclients 106 in these examples may be communicating with the Internetnetwork 104 via a wireless network 104. Those skilled in the art willrecognize, however, that the invention described herein may beimplemented utilizing any type of data communications channel.

The server 102 may include components such as a communications module110, a content change detector 112, a content generator 114, a contentencoder 116, and a smoothing module 118. The communications module 110may facilitate the server 102 in transmitting information to a client106 via network 104 as well as receiving information from client 106 vianetwork 104. The communications module 110 may use any type of protocolin its communications with clients 106. In one embodiment, the server102 may receive client information from clients 106 using communicationsmodule 110 and may transmit encoded image frames and/or compressioninformation to clients 106 using communications module 110. Clientinformation may include information identifying the client 106 or itsuser (or the user's preferences) as well as information relating to theimage that should be streamed to the client 106, such as the user'scurrent location or orientation. In one example, for an image streamingapplication that displays to a user a 3-dimensional (3-D) image based ontheir current position, client information may include an indication ofthe user's position, such as coordinates. For another application thatdisplays to a user a virtual rendering of what they should see, clientinformation may also include directional information such as a compassdirection that provides an indication of the direction the user islooking. Those skilled in the art will recognize that any type ofinformation may be included as client information.

The content generator 114 may generate a new image frame based on theclient information received by the communications module 110. Thecontent generator 114, which may be or utilize a multi-media orstreaming image application, may use any type of methodology todetermine the contents of a new image frame. The content generator 114may, for example, generate a new image frame based on the currentlocation or direction of the client 106 or its user, based on thereceived or calculated speed of the user, based on the occurrence of anevent, based on the current time, based on current weather or lightingconditions, based on the location of other clients 106 or users, or anyother factors. In one exemplary embodiment, content generator 114generates a new image frame to create a 3-D representation of thecurrent location of the client 106. In another exemplary embodiment,content generator 114 may generate a new image frame that depicts avirtual view of what a user with client 106 would see based on theircurrent location and direction. The content generator 114 in anotherembodiment may determine the velocity and direction of the client 106based on repeated receipt of client information and may then use thedetermined velocity and direction to generate an appropriate new imageframe. In yet another embodiment, the content generator 114 may receiveor capture an image and use that image to create an image frame forclient 106.

The content change detector 112 may analyze the new image framegenerated by the content generator 114 and compare it with the previousimage frame to determine if they differ so that the server 102 maychange the compression level in response to the changed content. Thecontent change detector 112 may use any type or combination ofmethodologies to determine if the content between a new image frame anda previous frame changed. In one embodiment, and as described in moredetail subsequently, the content change detector 112 may select a changedetection algorithm from a plurality of change detection algorithms. Inone embodiment, a user of the server 102 or of the client 106 may selecta change detection algorithm from a list of algorithms. In anotherembodiment, the content change detector 112 may select a changedetection algorithm based on factors such as server 102 performance orcapability, client 106 performance or capability, current network 104conditions, or network 104 capabilities. For example, if limitedcomputation resources were available for the content change detector 112it may choose a less resource-intensive algorithm. Example changedetection algorithms are described in relation to FIGS. 2, 4, and 5.

The content encoder 116 and smoothing module 118 may serve to, incombination or singly, determine the appropriate level of compressionfor the image frame to be transmitted to and encode the image frame atthe selected level of compression. In one embodiment, if there is nocontent change between successive image frames (as determined by thecontent change detector 112), the content encoder 116 may desire toimprove the quality of the image frame to be transmitted (and thus lowerthe compression level). In an interactive image streaming system, astatic image (not changing between successive frames) indicates that theimage from the client's point of view is stationary and image quality isthus raised in importance. In contrast, if successive images differ fromeach other, the client 106 point of view has motion and a lower qualityof image may be utilized (with higher compression levels). When a useris viewing an image depicting motion, details in the image may be lessimportant as details become blurred and the eye accounts for the loss indetail. When a user is viewing an image that is still, image details maybe more important as the user will be more likely to see any flaws orlack of details, compression artifacts, and the like. In one embodiment,the smoothing module 118 may vary the compression level via the lossyquantization step of compression standards such as Motion-JPEG or MPEGby using multiplier lookup tables for different levels of compression asrequired, though those skilled in the art will recognize that thesmoothing module 118 may use any type of smoothing mechanism.

The disclosed system may adapt the compression level based on the motiondepicted in the image stream (as represented by the difference, or lackthereof, between successive images) based on the perceived motion of theuser and/or client 106. As described in more detail in relation to FIG.3, the content encoder 116 may use the smoothing algorithms of thesmoothing module 118 to smooth the transition between different levelsof compression to improve the user experience. The smoothing module 118may determine a new level of compression for the image frame by runninga smoothing heuristic to generate a multi-frame smoothing routine. Whensuccessive images vary between each frame (implying motion), the contentencoder 116 may increase the compression level and reduce the bandwidthrequirements without detracting from the user's viewing experience. Whensuccessive images are the same (implying a stationary client 106), thecontent encoder 116 may decrease the compression level and increase thequality of the image. While such action may increase the bandwidthrequirements, streaming image quality for the user may be preserved. Inone embodiment, the content encoder 116 may not need to transmitsuccessive identical images and instead may transmit only an indicationto the use the previous image, saving bandwidth in the stationary usersituation. In an alternative embodiment, the content encoder 116 andsmoothing module 118 may base the selected compression level also on themagnitude or degree of change between frames as determined by thecontent change detector 112 so that slight changes to the image frame donot cause large jumps in the level of compression.

In the depicted embodiment, client 106 includes a communications module132, a decoder 134, a user display device 136, and a user input device138. The communications module 132 of the client 106, similarly to thecommunications module 110 of the server 102, may facilitate the client106 in transmitting information to server 102 via network 104 as well asin receiving information from server 102 via network 104 using any typeof protocol. In one embodiment, the client 106 may transmit clientinformation or an indication of a user selection of a change detectionalgorithm to server 102 using communications module 132 and may receiveencoded image frames and/or an indication of a compression level fromserver 102 using communications module 132. The decoder 134 may decodethe encoded image received by the communications module 132, such as bydecompressing the encoded image. The decoder 134 may use the indicationof the compression level received by the communications module 132 toassist it in decoding the image by accessing a multiplier lookup tablesassociated with the compression level. After the decoder 134 processesthe image, the user display device 136 may display the decoded image tothe user, such as by a display screen, printed images, holograms or anyother output device.

The user input device 138 may be any type of device to receive userinput, such as by keypad, keyboard, button, stylus, mouse, joystick,virtual reality device, voice command, or eye movement. User input mayinclude user preferences such as a selection of a change detectionalgorithm, a request for a higher or lower level of compression, arequest for faster screen rates, a user location or orientation, and thelike. Client 106 may also include devices that determine positioninformation such as a user location or orientation. Example devices mayinclude a Global Positioning System (GPS) receiver or otherlocation-determining device, electronic compass, or other devices. Theposition information may be a type of client information that allows theserver 102 in one embodiment to customize each generated image framebased on the current position of the client 106 and/or user.

The disclosed content change detector 112 may advantageously select orprovide a change detection algorithm for use in the interactive imagestreaming system 100. By allowing users to choose the change detectionalgorithm from a list or by selecting the change detection algorithmbased on current system 100 capabilities or conditions, the contentchange detector 112 may utilize the appropriate change detectionalgorithm. Using the selected change detection algorithm, the contentchange detector 112 may facilitate operation of the content encoder 116and smoothing module 118 to provide an effective mechanism fortransitioning compression levels with little impact on a viewer in animage streaming system such as the disclosed interactive image streamingsystem 100.

FIG. 2 depicts a content change detector of the interactive imagestreaming system of FIG. 1 according to one embodiment. In the depictedembodiment, the content change detector 112 includes a componentinterface module 202, an algorithm selector 204, and a plurality ofchange detection algorithm modules. The change detection algorithmmodules may include a compression size comparison algorithm module 206,a vector processing algorithm module 208, an indexed vector processingalgorithm module 210, a scene tracking algorithm module 212, and aclient information analysis algorithm module 214. Those skilled in theart will recognize that the content change detector 112 may include anytype, combination, and number of change detection algorithm modules. Thecontent change detector 112 may determine if a change has occurred or itmay alternatively also determine the magnitude or degree of the changebetween image frames. This may be useful when the magnitude or degree ofchange will have an impact on the chosen compression level.

The component interface module 202 may facilitate transmitting andreceiving information from other components of the server 102, such asthe content encoder 116, smoothing module 118, or communications module110. The algorithm selector 204 may select the change detectionalgorithm for use in analyzing and compressing an image stream. Asdescribed in more detail in relation to FIG. 5, the algorithm selector204 may use information such as a user selection or user preferences,system performance or capabilities, default settings, and the like inmaking its determination.

The compression size comparison algorithm module 206 may use acomparison of the compression sizes of each image frame to determinewhether a change has occurred. The algorithm may compress the imageframes itself or receive compressed versions of the image frames. Anon-changing compressed image frame size in an interactive system likelyindicates that the client 106 is requesting the same image repeatedlyand the successive image frames are the same. While this algorithmignores the possibility that two different image frames are the samesize when compressed, in practice it may be an accurate indicator ofchanging image streams as two completely different consecutive imageframes are very unlikely to have the same compressed image frame size.The compression size comparison algorithm module 206 may use anymethodology to determine the size of the image frame, such as vector summethodologies. This algorithm may be particular beneficial for servers102 that are memory-constrained in capabilities more than they areprocessor-constrained.

The vector processing algorithm module 208 may utilize vector orstatistic processing of bit maps of two successive image frames by usingBoolean analysis to determine if there is a difference between vectors.To accomplish this, the vector processing algorithm module 208 mayperform a Boolean sum check across each line of the image frames. Thisalgorithm will result in a computation of changes between the images andthe location of the changes. The vector processing algorithm module 208may process the sum check relatively quickly, making this algorithmparticular suited for processor-constrained environments. The indexedvector processing algorithm module 210 may perform an indexed version ofthe vector or statistic processing algorithm when the image frames havemore regularity and speed is more critical.

The scene tracking algorithm module 212 may use scene tracking or otheralgorithms to attempt to predict the level at which the user of theclient 106 is interacting with her environment, using the prediction tohelp determine whether a new image frame is the same as a previous imageframe. In one embodiment, the scene tracking algorithm module 212 mayuse information relating to a user is interacting with an application toanticipate what the user will do next or to gauge their level ofinteraction. A user who is barely interacting with an application, forexample, may be more likely to have a content change in successiveframes of an interactive application. The scene tracking algorithm mayprovide a high level of performance but may also large computationalrequirements and be more difficult to generalize or extend to otherapplications.

The client information analysis algorithm module 214 may use clientinformation to determine if a change in an image frame has occurredusing any methodology. For example, the client information analysisalgorithm module 214 may determine for a location-based application thatif the user's location has not changed (based on the clientinformation), the image frame to be next displayed to them need notchange, either. The client information analysis algorithm may beparticularly useful for certain types of applications, such aslocation-based applications, but may not have applicability to othertypes of applications.

FIG. 3 depicts an example of a flow chart for generating, encoding, andtransmitting a new image frame according to one embodiment. The server102 or any of its components, alone or in combination, may perform themethod of flow chart 300. Flow chart 300 begins with element 301,selecting a change detection algorithm. At element 301, and as describedin more detail in relation to FIG. 5, the content change detector 112may select a change detection algorithm based on user input, systemcapabilities or loads, or other methodologies. After the content changedetector 112 selects a change detection algorithm, the method of flowchart 300 continues to element 302, where the communications module 110may receive client information from a client 106 via network 104. Afterreceiving the client information, the content generator 112 may generatea new image frame based on the received client information at element304. The content generator 112 may generate the new image frame usingany type of algorithm depending on the application. For example, thecontent generator 112 may generate an image frame representing a viewfrom the user's point of view if the application so required.

The smoothing module 118 may next determine a starting compression levelfor the generated image frame at element 306. In one embodiment, thesmoothing module 118 may simply select the most recently-usedcompression level as the starting compression level. In anotherembodiment, the smoothing module 118 may base the starting compressionlevel on whether the system is within a multi-frame smoothing routine.While within a multi-frame smoothing routine, the starting compressionlevel may be based on the compression level appropriate for the currentframe of the multi-frame smoothing routine. In one example, of thesmoothing module 118 started a compression routine that started at acompression level of five (5) and ended at ten (10) over six (6) frames,incrementing one compression level per frame, the starting compressionlevel for the third frame may be compression level seven (7) (5+2=7). Inone embodiment, the smoothing module 118 may utilize a multi-framesmoothing routine that extends for one to two seconds in order to reducethe impact on the user. In one example, if the image stream was beingtransmitted at 15 frames per second a multi-frame smoothing routine of15 to 30 frames to transition from one compression level to another mayprove useful. Those skilled in the art will recognize that the smoothingmodule 118 may utilize other multi-frame smoothing routines.

The content change detector 112 may compare the generated new image tothe previous image at element 308 after the smoothing module 118determines the starting compression level using the change detectionalgorithm it selected at element 301. The results of the imagecomparison may include an indication that there was a change in contentbetween the frames, an indication that there was no change in content,or an indication of the degree or amount of change. If the contentchange detector 112 determines at decision block 310 that the contenthas changed between the generated new image and the previous imageframe, the method of flow chart 300 continues to element 314. At element314, the smoothing module 118 may perform a smoothing heuristic on thenew image to decrease its quality and increase the level of compression.The smoothing module 118 increases the level of compression as thecontent change between images indicates that the user's perspective islikely moving and that the user is thus likely to tolerate lower imagequality. If the content change detector 112 determines at decision block310 that the content has not changed between the generated new image andthe previous image frame, the method of flow chart 300 continues toelement 312. At element 312, the smoothing module 118 may perform asmoothing heuristic on the new image to increase its quality anddecrease the level of compression. The smoothing module 118 may decreasethe level of compression as the lack of content change between imagesindicates that the user's perspective is likely static and that the useris thus likely to require higher image quality.

The smoothing module 118 may perform its smoothing heuristic at elements312 or 314 to change the compression level between an initialcompression level and a target compression level using any methodology.In one embodiment, the smoothing module 118 may increment between theinitial compression level and the target compression level (in theappropriate direction depending on whether it is increasing ordecreasing) through a series of compression quality steps resulting in amulti-stage compression routine over multiple image frames. For example,the smoothing module 118 may use five levels of compression between theminimum and maximum levels of compression. For each level ofcompression, however, the server 102 and client 106 may each need tocontain a multiplier lookup table for that level of compression. Thenumber of compression levels requires a trade-off between memory usageat both the server 102 and client 106 and the smoothness of theheuristic and the resulting streaming images. A higher number ofcompression levels may result in a smoother transition betweencompression levels but may require additional amounts of memory, whichmay be problematic particularly for a client 106 that may have limitedprocessing and memory resources. Three or more levels of compression aretypically desirable for smooth transitions but one skilled in the artwill recognize that lower levels are possible with potentially degradedperformance. The smoothing module 118 may, in generating a multi-framesmoothing routine, use a roughly linear transition from one compressionlevel to another or may alternatively use any other type of algorithm,such as one biased towards higher or lower compression levels.

After the smoothing module 118 has performed its smoothing heuristic,the content encoder 116 may set the new compression level for the imageframe at element 316 based on the results from the smoothing module 118.With the new compression level the content encoder 116 may then encodethe new image frame based on the new compression level at element 318.As described previously, the content encoder 116 may use any type ofencoding algorithm to encode the new image frame. The communicationsmodule 110 may then transmit the encoded new image frame and anindication of the new compression level to the client 106 at element320, after which the method of flow chart 300 terminates. The method offlow chart 300 may also return to element 302 to repeat the process foreach frame (and may skip element 301 during normal operation).

FIG. 4 depicts an example of a flow chart for receiving user input andreceiving and displaying an image frame to a user according to oneembodiment. The client 106 or any of its components, alone or incombination, may perform the method of flow chart 400. Flow chart 400begins with element 402, displaying a list of change detectionalgorithms. At element 402, the user display device 136 may display alist of any available change detection algorithms to a user so that theymay choose from the list. At element 404, the user input device 138 mayreceive an indication of the user's selection, such as by a userselecting an algorithm from the displayed list. While the selection of achange detection algorithm by the user is described as the userselecting an algorithm from a list, other methodologies are alsopossible. The client 106 may, for example, either explicitly ask theuser questions or infer preferences that are implicit from the user'sbehavior. The client 106 may make its determination based on directfeedback (questions or inferences based on the current stream) orindirect feedback. Indirect feedback may include determinations based oncomparisons to other aspects, such as past images or streams, relatedapplications, similar users, or similar hardware.

Examples of explicit or implicit queries and direct or indirect feedbackmay provide further illumination. In one example, the client 106 may askdirect and explicit questions of the user to determine their selectionby asking the user if, based upon the stream, the settings should bechanged up or down. An explicit question related to the current streammay provide an accurate portrayal of the user's preferences but doesrequire the user to respond (and thus may impair the user's experience).In another example, the client 106 may present the user with choices ofdifferent frames or streams before streaming images showing differencesbetween compression algorithms from which they may select. In anotherexample, the client 106 may infer user preferences that are implicit intheir behavior, such as by inferring that a user finds a particularcompression level acceptable based on the user's continued viewing ofthe image stream. One skilled in the art will recognize that the client106 may utilize other methodologies for determining a selected algorithmin addition to those described herein.

The communications module 132 may transmit the indication of the user'salgorithm selection at element 406. Client 106 may utilize elements 402,404, and 406 in an embodiment where the user has a choice of changedetection algorithms and may execute the elements before or during thereceipt of streaming image content (such as an in an interrupt routine).In one embodiment, client 106 may execute elements 402, 404, and 406before an image stream begins so that the elements need not be repeatedfor each frame of the image stream.

After optionally processing the user's selection of an algorithm, flowchart 400 continues to element 408, receiving user input relevant to thenext frame. In one embodiment of element 408, the communications module132 may receive input from a user via user input device 138, asdescribed previously, or it may receive input from a different devicesuch as a location determining device. The user input device 138 mayoptionally process the user input at element 410, such as by convertingthe input to another form for transmission. At element 412, thecommunications module 132 may next transmit an indication of the userinput to the server 102 via network 104.

After transmitting an indication of the user input, the method of flowchart 400 may continue to element 414, where the communications module132 may receive an encoded image frame from the server 102 via network104. At element 416, the communications module 132 may also receive anindication of the compression level for the encoded image frame from theserver 102 via network 104. In one embodiment, the communications module132 may receive the encoded image frame and the indication of thecompression level in the same transmission.

Using the indication of the compression level, the decoder 134 maydecode the encoded image frame at element 418. The decoder 134 may use alookup table related to the indicated compression level to perform adequantization step of the decoding process in one embodiment. After theimage frame is decoded, the user display device 136 may display thedecoded image frame to the user at element 420, after which the methodof flow chart 400 terminates. The method of flow chart 400 may alsoreturn to element 408 to repeat the entire flow chart for each frame(without having to repeat the user selection of an algorithm at elements402, 404, 406).

FIG. 5 depicts an example of a flow chart for selecting a changedetection algorithm and comparing the content of successive image framesaccording to one embodiment. One or more components of the contentchange detector 112 may perform the method of flow chart 500 in oneembodiment, such as during the execution of element 301 of FIG. 3. Flowchart 500 begins with element 502, where the algorithm selector 204 mayoptionally set the change detection algorithm to a default setting. Thealgorithm selector 204 may set the default based on user input, based onthe most recently used change detection algorithm, or any othermethodology.

At decision block 504, the algorithm selector 204 may determine whethera user has made a selection of a preferred change detection algorithm.If the user has made a selection, flow chart 500 continues to element508, where the algorithm selector 204 may receive an indication of theuser's selection, such as from the component interface module 202. Thealgorithm selector 204 may then select the change detection algorithmbased on the user's selection at element 510. As described in relationto FIG. 4, the user's selection may include an indication of a user'sspecific selection of an algorithm, an indirect indication of the user'spreference, an inferred indication of the user's preference, or otherinformation. Alternatively, the algorithm selector 204 may ignore theuser's request and leave the change detection algorithm as a default ormay select it based on other factors, such as when the change detectionalgorithm selected by the user is unsuitable for some reason.

If the user has not made a selection (or if their selection is beingoverridden), flow chart 500 continues to element 512, where thealgorithm selector 204 may receive an indication of system capabilities,such as from the component interface module 202. System capabilities mayinclude performance or capability information about the server 102,network 104, and/or client 106, such as computational or processingpower, storage space, memory, current loads, bandwidth, or the like. Atelement 514, the algorithm selector 204 may select a change detectionalgorithm for the available algorithms based on the received informationabout system capabilities. For example, if the system capabilitiesindicate that server 102 processing power is limited (such as if theserver 102 were a handheld wireless device), the algorithm selector 204may select the compression size comparison algorithm as it is lessprocessor-intensive than some other algorithms. In another example, thealgorithm selector may select the vector processing algorithm in aspeed-constrained environment. In another example, the algorithmselector 204 may shift processing to the server 102 if the client 106has a large bandwidth pipeline but little computational power.Similarly, if the client 106 has little bandwidth but more computationalpower, the algorithm selector 204 may shift some or all of theprocessing to the client 106. Those skilled in the art will recognizethat the algorithm selector may utilize any suitable methodology todecide between algorithms. After the algorithm selector 204 selects achange detection algorithm at element 510 or 514, as appropriate, themethod of flow chart 500 terminates.

In general, the routines executed to implement the embodiments of theinvention, may be part of an operating system or a specific application,component, program, module, object, or sequence of instructions. Thecomputer program of the present invention typically is comprised of amultitude of instructions that will be translated by the native computerinto a machine-readable format and hence executable instructions. Also,programs are comprised of variables and data structures that eitherreside locally to the program or are found in memory or on storagedevices. In addition, various programs described hereinafter may beidentified based upon the application for which they are implemented ina specific embodiment of the invention. However, it should beappreciated that any particular program nomenclature that follows isused merely for convenience, and thus the invention should not belimited to use solely in any specific application identified and/orimplied by such nomenclature.

It will be apparent to those skilled in the art having the benefit ofthis disclosure that the present invention contemplates methods,systems, and media for detecting content change in a streaming imagesystem. It is understood that the form of the invention shown anddescribed in the detailed description and the drawings are to be takenmerely as examples. It is intended that the following claims beinterpreted broadly to embrace all the variations of the exampleembodiments disclosed.

1. A method for detecting a content change between image frames in astreaming image system, the method comprising: selecting by a contentchange detector a change detection algorithm from a plurality of changedetection algorithms; comparing by the content change detector a firstimage frame of an image stream with a second image frame of the imagestream using the selected change detection algorithm; and in the eventof detecting a content change between the first image frame and thesecond image frame, generating by the content change detector anindication of the content change.
 2. The method of claim 1, furthercomprising transmitting by the content change detector the indication ofthe content change.
 3. The method of claim 1, further comprising:receiving by the content change detector an indication of a user'sselection of a change detection algorithm; and wherein selecting by thecontent change detector the change detection algorithm from a pluralityof change detection algorithms comprises selecting by the content changedetector the change detection algorithm based on the user's selection.4. The method of claim 1, further comprising: receiving by the contentchange detector an indication of system capabilities; and whereinselecting by the content change detector the change detection algorithmfrom a plurality of change detection algorithms comprises selecting bythe content change detector the change detection algorithm based on thesystem capabilities.
 5. The method of claim 1, wherein selecting by thecontent change detector the change detection algorithm from a pluralityof change detection algorithms comprises selecting by the content changedetector a default change detection algorithm.
 6. The method of claim 1,wherein the plurality of change detection algorithms comprises one ormore of a compression size comparison algorithm, a vector processingalgorithm, an indexed vector processing algorithm, or a scene trackingalgorithm.
 7. The method of claim 1, wherein one of the plurality ofchange detection algorithms is a compression size comparison algorithm,and wherein the content change detector is adapted to apply thecompression size comparison by comparing a compressed first image frameand a compressed second image frame.
 8. A machine-accessible mediumcontaining instructions effective, when executing in a data processingsystem, to cause said data processing system to perform operationscomprising: selecting by a content change detector a change detectionalgorithm from a plurality of change detection algorithms; comparing bythe content change detector a first image frame of an image stream witha second image frame of the image stream using the selected changedetection algorithm; and in the event of detecting a content changebetween the first image frame and the second image frame, generating bythe content change detector an indication of the content change.
 9. Themachine-accessible medium of claim 8, further comprising transmitting bythe content change detector the indication of the content change. 10.The machine-accessible medium of claim 8, further comprising: receivingby the content change detector an indication of a user's selection of achange detection algorithm; and wherein selecting by the content changedetector the change detection algorithm from a plurality of changedetection algorithms comprises selecting by the content change detectorthe change detection algorithm based on the user's selection.
 11. Themachine-accessible medium of claim 8, further comprising: receiving bythe content change detector an indication of system capabilities; andwherein selecting by the content change detector the change detectionalgorithm from a plurality of change detection algorithms comprisesselecting by the content change detector the change detection algorithmbased on the system capabilities.
 12. The machine-accessible medium ofclaim 8, wherein selecting by the content change detector the changedetection algorithm from a plurality of change detection algorithmscomprises selecting by the content change detector a default changedetection algorithm.
 13. The machine-accessible medium of claim 8,wherein the plurality of change detection algorithms comprises one ormore of a compression size comparison algorithm, a vector processingalgorithm, an indexed vector processing algorithm, or a scene trackingalgorithm.
 14. The machine-accessible medium of claim 8, wherein one ofthe plurality of change detection algorithms is a compression sizecomparison algorithm, and wherein the content change detector is adaptedto apply the compression size comparison by comparing a compressed firstimage frame and a compressed second image frame.
 15. A content changedetector system, the system comprising: a component interface module,the component interface module being adapted to receive an image frameto be compared; a plurality of change detection modules, each of thechange detection modules having an associated change detectionalgorithm, the plurality of change detection modules being adapted tocompare the received image frame with a previous image frame using itsassociated change detection algorithm; and an algorithm selector incommunication with the component interface module, the algorithmselector being adapted to select one of the plurality of changedetection algorithms for comparing the received image frame with theprevious image frame.
 16. The system of claim 15, wherein the componentinterface module is further adapted to receive an indication of one ormore of a user selection of a change detection algorithm or anindication of system capabilities.
 17. The system of claim 15, whereinthe component interface module is further adapted to transmit anindication of comparison results of the image frame to be compared andthe previous image frame.
 18. The system of claim 15, wherein theplurality of change detection modules comprise one or more of acompression size comparison algorithm module, a vector processingalgorithm module, an indexed vector processing algorithm module, a scenetracking algorithm module, and a client information analysis algorithmmodule.
 19. The system of claim 15, wherein the algorithm selectorselects one of the plurality of change detection algorithms based on anindication of a user's selection of a change detection algorithm. 20.The system of claim 15, wherein the algorithm selector selects one ofthe plurality of change detection algorithms based on an indication ofsystem capabilities.